It is known that there are methods and models to track a three dimensional object in an environment and compute its position and orientation (pose) with respect to a predetermined coordinate system. These kinds of tracker systems are used, for example, in aircrafts to determine the orientation of a pilot's head. Once the orientation is acquired with respect to the coordinate system of a display devices, for example, then it is possible to generate graphics on these accordingly. There are different methods of tracking an object in a scene using magnetic, mechanical, or optical means. Currently, the spatial relations of objects may also be determined using magnetic sensors or laser beams, but this invention relates specifically to systems using camera-based (day-tv, thermal, IR, Time of Flight, etc.) trackers.
In one of the optical camera-based systems, the pilot wears a helmet with patterns (fiducial markers) and at least one tracker camera determines the helmet's position and orientation using geometric calculations based on these patterns. The patterns used in camera-based tracker systems are either graphical (generally black and white) patterns (passive marker) tracked by visible light cameras or arrays of light sources (e.g., light emitting diodes or LEDs) (active marker). These light sources can be chosen to be in the infrared range of the electromagnetic spectrum with suitable selection of camera sensor and filter set. Other arrangements are also possible but the most convenient among them is the one with the infrared LEDs since these systems can work in poor lighting conditions. Computing spatial relation between an object having a tracking pattern and a camera is, therefore, well known in the state of the art. Throughout the document, whenever a spatial relation is mentioned, it should be understood that the relation in question is between an entity's predetermined reference system with respect to the other's. The objective is to find rotation and translation between camera and 3D object so that the object's 3D location and orientation is known. This reference system is generally based on the respective pattern of an object under consideration. Since the position and orientation of the tracker camera with respect to the other coordinate systems is known (or can be calculated or measured) in a tracker system, it is also possible to compute the helmet's spatial relation with the tracker camera's sensor and then with other coordinate systems. In this context, “tracked object” means an object having a tracking pattern (fiducial marker) and being tracked by a tracker system. It may be either a helmet as in a helmet-mounted tracker system or any other object.
A pose estimation problem for said optical camera-based tracking systems can be stated as follows: Given a set of N feature correspondences between three-dimensional (3D) points of an object and two-dimensional (2D) projection of that object onto the image plane, find the rotation and translation of the object with respect to the reference system of the camera. The objective is to find rotation and translation between camera and 3D object so that the object's 3D location and orientation is known. Pose estimation problem requires correct solution to the correspondence problem. A correspondence problem can be stated as follows: A given (or not-given) rotation and translation of the object with respect to the reference system of the camera, find the N feature correspondences between 3D points of an object and 2D projection of that object on the image plane. The objective is to find correspondences between 3D points of an object and 2D projection of that object on the image plane so that this can be used to find the object's 3D location and orientation. This problem is also well known in the state of the art. However, we have two problems which each require a solution to the other problem to be correctly solvable. Thus, a tracker system requires an initial solution to the either pose estimation problem or the correspondence problem.
A correspondence problem is called an initial correspondence problem if rotation and translation of the object with respect to the reference system of the camera is not given. If one opts to solve the initial correspondence problem, then he/she also makes the completion time for solving correspondence problem as short as possible. Such an effort is necessary since it is possible that a tracked object may leave the view frustum of the camera then return to view. In such cases, restarting of the tracking process is necessary, thus solving the initial correspondence problem quickly and shortening the startup time. In one of the preferred applications, for helmet tracking, a fast startup time significantly reduces blind time, especially if a pilot's head leaves the cameras view frustum many times.
There are some methods currently used to determine initial correspondences between 3D points of the tracked object and 2D projection of that object on the image plane. In one of the methods, a number of consecutive images are used where a number of lit LEDs increase by one at each iteration. Then with proximity calculations, 3D to 2D matching is calculated with the assumption that pose change won't be that great between consecutive captured frames. A newly lit LED will be the one that has not matched to any 2D point, which is then added to the matched list. Solving a correspondence problem is easier where the pose of the tracked object is known with small error. Then similar proximity calculations can be carried out to find 3D to 2D matching.
Solving initial correspondence problems with the current methods require many frame captures, thus it requires a long time, whereas usage of methods that use pose of the tracked object can't be used since rotation/translation data of tracked object is not initially available. The current methods do not offer an effective way of solving the initial correspondence problem. To provide a solution to this problem, a new methodology should be introduced which solves the problem in a very efficient way. Furthermore, the proposed method can be used to solve the correspondence problem (not only the initial case) thus offering an all-around solution to the correspondence problem.
With the proposed method, all the LEDs used for pose determination can be identified. And also the proposed method comprises a series of steps regarding how to select an LED group to be used for solving correspondence problem between the LED group's 3D location and the LED group's respective 2D projections.
The United States patent document US005828770A (Lets, Ristau), an application in the state of the art, discloses a method based on proximity calculations which uses a number of consecutive frame captures to solve the initial correspondence problem. In the same document, a similar proximity based method solves the correspondence problem when the pose of the tracked object is known is also presented.
Boschetti et al., 2005, deals with the problem of guiding an operator during the execution of a 5 degrees of freedom surgical teleoperation task. The work is focused on the feasibility of a spine surgery telerobotic system, made up of a haptic master, a slave robot, an optical tracking device and a main control unit. With this system, the surgeon performs a drilling operation by using a telerobotic device, being guided by haptic feedback: as soon as the vertebra moves, the tracking device measures vertebra pose and a proper force is exerted on the operator's hand to let him/her adjust surgical tool position and orientation. Moreover, the haptic master produces force feedback related to the teleoperation. The paper is focused on the design and implementation of this haptic system, with particular reference to control system architecture.
Dario et al., 2000 Describes a mechatronic tool for arthroscopy, which is both a smart tool for traditional arthroscopy and the main component of a system for computer-assisted arthroscopy. The mechatronic arthroscope, which has a cable-actuated servomotor-driven multi-joint mechanical structure, is equipped with a position sensor for measuring the orientation of the tip and a force sensor for detecting possible contact with delicate tissues in the knee and incorporates an embedded microcontroller for sensor signal processing, motor driving, and interfacing with the surgeon and/or the system control unit. The computer-assisted arthroscopy system comprises an image processing module for the segmentation and 3D reconstruction of pre-operative CT or MR images, a registration module for measuring the position of the knee joint, tracking the trajectory of the operating tools and matching pre-operative and intra-operative images, and a human-machine interface that displays the enhanced reality scenario and data from the mechatronic arthroscope in a friendly and intuitive manner. By integrating pre-operative and intra-operative images and information provided by the mechatronic arthroscope, the system allows virtual navigation in the knee joint during the planning phase and computer guidance by augmented reality during the intervention. This paper describes in detail the characteristics of the mechatronic arthroscope and of the system for computer-assisted arthroscopy and discusses experimental results obtained with a preliminary version of the tool and of the system.
Hoff/Vincent, 2000, discloses a method developed to analyze the accuracy of the relative head-to-object position and orientation (pose) in augmented reality systems with head-mounted displays. From probabilistic estimates of the errors in optical tracking sensors, the uncertainty in head-to-object pose can be computed in the form of a covariance matrix. The positional uncertainty can be visualized as a 3D ellipsoid. One useful benefit of having an explicit representation of uncertainty is that we can fuse sensor data from a combination of fixed and head-mounted sensors in order to improve the overall registration accuracy. The method was applied to the analysis of an experimental augmented reality system, incorporating an optical see-through head-mounted display, a head-mounted charge-coupled device (CCD) camera, and a fixed optical tracking sensor. The uncertainty of the pose of a movable object with respect to the head-mounted display was analyzed.